Brennan Center

Some good stuff from the DMN.

When Social Sentinel representatives pitched their service to Florida’s Gulf Coast State College in 2018, they billed it as an innovative way to find threats of suicides and shootings posted online. But for the next two years, the service found nothing dangerous.

One tweet notified the school about a nearby fishing tournament: “Check out the picture of some of the prizes you can win – like the spear fishing gun.”

Another quoted the lyrics from a hit pop song from 2010: “Can we pretend that airplanes in the night sky are like shooting stars? I could really use a wish right now.”

As police and administrators fielded a flood of alerts about posts that seemed to pose no threat, the company told the school in emails that it had eliminated more than half of all irrelevant alerts. Months later, they said the number had decreased by 80%. By January 2019, the company told schools its service flagged 90% fewer irrelevant posts.

But at Gulf Coast, the problem continued.

One alert from March 2019 read, “Hamburger Helper only works if the hamburger is ready to accept that it needs help.”

“Nothing ever came up there that was actionable on our end,” David Thomasee, the executive director of operations at Gulf Coast, said in an interview earlier this year. The college stopped using the service in April 2021.

Gulf Coast was not the only college inundated with irrelevant alerts. Officials from 12 other colleges raised concerns about the performance of Social Sentinel in interviews and emails obtained by The Dallas Morning News and the Investigative Reporting Program at UC Berkeley’s Graduate School of Journalism.

Only two of the 13, North Central Texas College and the University of W Connecticut, still use the service.

As schools and universities confront a worsening mental health crisis and an epidemic of mass shootings, Social Sentinel offers an attractive and low-cost way to keep students safe. But experts say the service also raises questions about whether the potential benefits are worth the tradeoffs on privacy.

Records show Social Sentinel has been used by at least 38 colleges in the past seven years, including four in North Texas. The total number is likely far higher — The company’s co-founder wrote in an email that hundreds of colleges in 36 states used Social Sentinel.

The News also analyzed more than 4,200 posts flagged by the service to four colleges from November 2015 to March 2019. None seem to contain any imminent, serious threat of violence or self-harm, according to a News
analysis, which included all of the posts obtained through public records requests.

Some schools contacted by The News said the service alerted them to students struggling with mental health issues. Those potential success stories were outweighed by complaints that the service flagged too many irrelevant tweets, interviews and emails between officials show. None of the schools could point to a student whose life was saved because of the service.

[…]

For one former Social Sentinel employee, it only took three days before they had serious doubts about the effectiveness of the service.

The worker estimated that 99.9% of the flagged posts sent to clients were not threatening. The service often crashed because it flagged too many posts. At least 40% of clients dropped the service every year, the employee said.

Over the course of several months, the employee repeatedly raised concerns with supervisors and fellow employees about flaws in the system, but those complaints were often ignored, the worker said.

The employee, who asked not to be named for fear of retribution, said problems with the service were an open secret at the company, and described it as “snake oil” and “smoke and mirrors.”

The News also contacted more than two dozen other former company employees, who either did not respond or said they had signed nondisclosure agreements preventing them from speaking publicly about their time at the company.

At the University of Texas at Dallas, which started using the service in 2018, campus police officers in charge of the service also grew increasingly skeptical of its performance, emails obtained through a records request show.

“Does the company have any data (not anecdotal) to show its success rate in mitigating harm or disaster through its alert system?” UT Dallas Police Lieutenant Adam Perry asked his chief in an email obtained by The News. The chief forwarded the email to a company employee who didn’t answer the question.

Perry said that while the school used the service, the technology never alerted police to legitimate threats of suicide or shootings.

“I think in concept, it’s not a bad program,” Perry said. “I just think they need to work on distinguishing what a real threat is.” UT Dallas ended its use of the service last year.

Ed Reynolds, police chief at the University of North Texas, defended the system, but also estimated that “99.9 percent (of the alerts) were messages we didn’t need to do anything with.” After using the service for about three years, UNT ended its contract with the company in November 2018.

As noted before, the Uvalde school district was among the ISDs in Texas that have used Social Sentinel. Putting my cybersecurity hat on for a minute, there are similar services in that space that do provide good value, but they have been around longer, there’s far more data on cyber threats, and it’s much easier to configure alerts for these services to very specific things, which greatly reduces the noise factor. I do think a service like this could be useful, but what we have now is not mature enough. More data and more analysis to help eliminate likely false positives before they show up in a customer’s alert feed are needed. Even with that, it’s still likely to be noisy and to require fulltime human analysis to get value out of it. For now, the best use of this is probably for academics. After they’ve had some time with it, then school districts and colleges might make use of it.

While current Republican “solutions” for gun violence include door control and arming teachers, one “solution” that has been in place for the past few years has been monitoring social media for signs of gun-related threats. That was in place in Uvalde, and it was not effective.

After a shooter killed 21 people, including 19 children, in the massacre at Robb Elementary School in Uvalde, Texas, last week, the United States is yet again confronting the devastating impact of gun violence. While lawmakers have so far failed to pass meaningful reform, schools are searching for ways to prevent a similar tragedy on their own campuses. Recent history, as well as government spending records, indicate that one of the most common responses from education officials is to invest in more surveillance technology.

In recent years, schools have installed everything from facial recognition software to AI-based tech, including programs that purportedly detect signs of brandished weapons and online screening tools that scan students’ communications for mentions of potential violence. The startups selling this tech have claimed that these systems can help school officials intervene before a crisis happens or respond more quickly when one is occurring. Pro-gun politicians have also advocated for this kind of technology, and argued that if schools implement enough monitoring, they can prevent mass shootings.

The problem is that there’s very little evidence that surveillance technology effectively stops these kinds of tragedies. Experts even warn that these systems can create a culture of surveillance at schools that harms students. At many schools, networks of cameras running AI-based software would join other forms of surveillance that schools already have, like metal detectors and on-campus police officers.

“In an attempt to stop, let’s say, a shooter like what happened at Uvalde, those schools have actually extended a cost to the students that attend them,” Odis Johnson Jr, the executive director of the Johns Hopkins Center for Safe and Healthy Schools, told Recode. “There are other things we now have to consider when we seek to fortify our schools, which makes them feel like prisons and the students themselves feel like suspects.”

[…]

Even before the mass shooting in Uvalde, many schools in Texas had already installed some form of surveillance tech. In 2019, the state passed a law to “harden” schools, and within the US, Texas has the most contracts with digital surveillance companies, according to an analysis of government spending data conducted by the Dallas Morning News. The state’s investment in “security and monitoring” services has grown from $68 per student to $113 per student over the past decade, according to Chelsea Barabas, an MIT researcher studying the security systems deployed at Texas schools. Spending on social work services, however, grew from $25 per student to just $32 per student during the same time period. The gap between these two areas of spending is widest in the state’s most racially diverse school districts.

The Uvalde school district had already acquired various forms of security tech. One of those surveillance tools is a visitor management service sold by a company called Raptor Technologies. Another is a social media monitoring tool called Social Sentinel, which is supposed to “identify any possible threats that might be made against students and or staff within the school district,” according to a document from the 2019-2020 school year.

It’s so far unclear exactly which surveillance tools may have been in use at Robb Elementary School during the mass shooting. JP Guilbault, the CEO of Social Sentinel’s parent company, Navigate360, told Recode that the tool plays “an important role as an early warning system beyond shootings.” He claimed that Social Sentinel can detect “suicidal, homicidal, bullying, and other harmful language that is public and connected to district-, school-, or staff-identified names as well as social media handles and hashtags associated with school-identified pages.”

“We are not currently aware of any specific links connecting the gunman to the Uvalde Consolidated Independent School District or Robb Elementary on any public social media sites,” Guilbault added. The Uvalde gunman did post ominous photos of two rifles on his Instagram account before the shooting, but there’s no evidence that he publicly threatened any of the schools in the district. He privately messaged a girl he did not know that he planned to shoot an elementary school.

Any kind of surveillance involves a tradeoff between privacy and security. So far, the security gains from software like this are small, while the loss of privacy – which to be clear here is the privacy of children – is significant.

For privacy advocates, the lack of evidence for the technology’s effectiveness means that there are no sufficient grounds for the potential violations of privacy that come with its use. Hye Jung Han, a researcher at Human Rights Watch specializing in child rights, told The Verge that using surveillance technology on children could cause unwarranted harm:

“Could you imagine schools using toxic materials to build classrooms, even if it hadn’t met any safety standards? No,” said Han. “Similarly, to use unproven, untested surveillance technologies on children, without first checking whether they are safe to use, exposes children to an unacceptable risk of harm.”

Multiple requests for comment sent to Navigate360 — which acquired Social Sentinel in 2020 — did not receive a response.

The Uvalde school district was confirmed to have purchased monitoring capability from Social Sentinel in 2019–2020, though it is unclear whether the subscription was still active at the time of the shooting. However, even if it had been, the technology would have been unlikely to flag any of the shooter’s posts. There are now numerous reports of concerning activity surrounding the shooter’s online activity: he allegedly made frequent threats to young women and girls via chat apps, sent images of guns to acquaintances, and reportedly discussed carrying out the school shooting in an Instagram chat. But Social Sentinel is only able to monitor public posts and would not have had access to any content shared in private messages.

At the same time, there are significant privacy concerns with the software. In 2019, the Brennan Center for Justice outlined a range of civil and human rights concerns stemming from expanded social media monitoring in K-12 schools, among them the questionable effectiveness of the technology in combination with a tendency to disproportionately impact students from minority communities. In the same year, reporting by Education Week also covered the dramatic expansion of digital surveillance in schools, highlighting the large number of false positives generated by Social Sentinel’s technology. (Alerts were reportedly triggered by tweets about the Mark Wahlberg movie, Shooter and from a student pleased their credit score was “shooting up,” among other things.)

Of all US states, Texas has been the most enthusiastic about the use of digital surveillance for school children. A 2021 investigation by The Dallas Morning News found that no state has more school districts contracting with digital surveillance companies than Texas. But of the Texas districts that did take out these contracts, results were apparently mixed: a number of school districts that had paid for Social Sentinel told the Morning News that they had declined to renew contracts, describing a service that provided few actionable alerts or flagged mostly irrelevant information.

But while Social Sentinel advertises an ability to monitor a broad range of platforms, there’s some suggestion that its surveillance capabilities are dictated more by the accessibility of data sources than by their importance. A client presentation from the company shared by the EFF lists a range of social media sources for monitoring, including Instagram, YouTube, Vimeo, Flickr, Tumblr, WordPress, and even Meetup.

Data obtained by BuzzFeed News confirmed this through data obtained under the Freedom of Information Act, which showed the company skewed heavily towards Twitter monitoring. Of the 1,206 Social Sentinel alerts provided to BuzzFeed, 98 percent (1,180) related to tweets — even though Instagram, YouTube, and even Facebook are more widely used by younger demographics. But the conventions of Twitter — where the vast majority of posts are publicly visible, even unintentionally — mean that it is comparatively easier to monitor, providing a wealth of social media data on tap that can be assimilated by companies looking to boost their surveillance credentials.

The DMN reports that some of the school districts that kicked the tires on Social Sentinel later decided it wasn’t worth it.

Uvalde is among at least 52 school districts and three colleges in Texas that have used the Social Sentinel service, according to records from GovSpend, an organization that tracks state and local government spending. It has also been used by dozens of colleges and hundreds of school districts nationwide.

Uvalde purchased Social Sentinel in August 2019, according to GovSpend. A document from the 2019-2020 school year lists the service as one of the district’s “preventative security measures.”

“UCISD utilizes Social Sentinel to monitor all social media with a connection to Uvalde as a measure to identify any possible threats that might be made against students and or staff within the school district,” the document reads.

The district made two payments to the company totaling more than $9,900, the data show.

Several Texas districts that have used Social Sentinel complained the service was mostly ineffective. The News reached out to every school district that used Social Sentinel, including Uvalde, for comment last year. Clear Creek ISD, a district outside of Houston, used the service in the 2018-19 school year but soon canceled.

“The Clear Creek Independent School District discontinued the use of Social Sentinel in its first year,” Elaina Polsen, Clear Creek’s chief communications officer, told The News last year. “The District determined the service just did not meet our needs, and we were receiving far stronger information through our anonymous tip line.”

Representatives from Keller, Lewisville, Mineral Wells and Schertz-Cibolo school districts also said the service provided them with few alerts or alerts that contained mostly irrelevant information.

HISD does not appear to have been a user of Social Sentinel, so we’ve got that going for us. There are other companies with similar products out there, so be on the lookout for that kind of pitch. It’s not out of the question to me that a tool like this could be effective at some point (we would still have to debate the privacy impact, and I can just about guarantee that it won’t be good), but we’re not there yet and it may be awhile before we can reasonably broach the subject. In the meantime, I dunno, maybe ban assault weapons again like we did in the 90s? Worked pretty well back then, and it didn’t involve snooping on things kids were saying among themselves. Just a thought.

(FYI, I first heard about Social Sentinel and its connection to Uvalde on the What Next podcast. I went looking for the DMN story from there, and found the others in the same search.)