The editors of Netology have prepared an adapted translation of the Wired article about this unknown to a wide circle of history, about the developments of Bledsoe and his team, which are used in modern face recognition technology.
For about thirty years Woody Bledsoe was a professor at the University of Texas at Austin and worked on the development of automated reasoning and artificial intelligence. According to the memoirs of Lance, Bledsoe's son, the professor was an enthusiastic optimistic scientist who, back in the late 1950s, dreamed of creating a computer endowed with human capabilities and capable of proving complex mathematical theorems, maintaining a conversation and playing decently ping-pong.
But early in his career, Bledsoe was keenly looking for an opportunity to teach machines to recognize faces - an underestimated but potentially powerful human ability. These were the first studies on face identification (1960), and the professor's work attracted the interest of the US intelligence services. Woody's main investors were likely CIA front companies.
:
Today, facial recognition is used to ensure security in phones, laptops, passports and payment applications. This technology is expected to revolutionize the targeted advertising market and accelerate the diagnosis of certain diseases. At the same time, facial identification technology is turning into an instrument of government pressure and corporate surveillance.
For example, with the help of this technology in China, the government is tracking representatives of the Uyghur ethnic minority, hundreds of thousands of whom were placed in political prison camps. And in the United States, according to The Washington Post, the Immigration and Customs Police and the FBI are conducting a digital search: looking for suspects in government databases of driver's licenses - sometimes without first going to court.
In 2019, a Financial Times investigation revealed that researchers from Microsoft and Stanford University collected and made publicly available a large number of data packets of images of people without the knowledge or consent of the photographed. Subsequently, this data was destroyed, but researchers of tech startups and one Chinese military academy managed to get it.
Woody Bledsoe's research on facial recognition in the 1960s anticipated the technological breakthroughs and ethical aspects we are seeing today. Yet these foundational works are almost entirely unknown - most of them have never been made public.
In 1995, for unknown reasons, Woody asked his son to destroy the research archive. But most of the papers survived, and thousands of pages of his work are now housed in the Briscoe Center for American History at the University of Texas.
Among other things, dozens of photographs of people have survived, and some faces are marked with strange mathematical notes, as if struck by some kind of "geometric" skin disease. In these portraits, one can discern the history of the emergence of technology, which in the coming decades will actively develop and penetrate into many areas of human activity.
How it all began. Tuple method
Woodrow Wilson (Woody) Bledsoe was born in 1921 into a large family of a sharecropper from Oklahoma. He was the tenth child in the family and as far as he could remember, he always helped his father with the housework. He possessed a mathematical mindset. Graduated from high school. He studied for three months at the University of Oklahoma, after which Woody was drafted into the army on the eve of World War II.
After the war, Woody studied mathematics at the University of Utah and then left for Berkeley to pursue his Ph.D. After graduating from graduate school, Woody worked in nuclear weapons research at the government-owned Sandia Corporation in New Mexico - along with luminaries such as Stanislaw Ulam, who helped create the hydrogen bomb.
At Sandia, Woody took the first steps in the computing world, a commitment to which will carry through his life. First I wrote the code for nuclear weapons projects. And later, Woody became interested in automatic pattern recognition, especially machine reading - the process of teaching the system to recognize unmarked images of written characters.
Woody Bledsoe and his colleague Iben Browning, an erudite inventor, aeronautical engineer and biophysicist, came up with a method that later became known as the n-tuple method.
Scientists began by projecting a printed symbol - say the letter Q - onto a rectangular grid of cells, like a lined sheet of paper. Each cell-cell was assigned a binary number depending on the presence or absence of a part of the symbol in it: 0 - for an empty cell, 1 - for a filled one. The cells were then randomly grouped into ordered pairs, like sets of coordinates. In theory, groups could include any number of cells, hence the name of the method. Then, using several mathematical steps, the system assigned a unique value to the symbol grid. And when a new symbol was encountered, the grid of that symbol was compared with others in the database until the nearest match was found.
The essence of the method was that it allowed recognizing many variations of the same sign: most Qs tended to get fairly similar results compared to other Qs. The process worked best with any pattern, not just text. According to Robert S. Boyer, mathematician and Woody's longtime friend, the tuple method helped define the scope of pattern recognition. This was one of the first steps to the question: "How can I program a machine to do what people do?"
Around the time he was developing the method of tuples, Woody first dreamed of creating a machine, which he called "computer man."
Years later, he recalled the “wild excitement” he felt when formulating skills for artificial intelligence:
“I wanted him to read typed characters and handwritten text. I could see him or part of him in a small camera that would be attached to my glasses, with an earpiece through which I would hear him call the names of my friends and acquaintances when I met them on the street ... You see, my computer friend could to recognize faces. "
Research at Panoramic Research Incorporated
In 1960, Woody - along with Iben Browning and another Sandia colleague - founded Panoramic Research Incorporated (Panoramic). They were initially housed in a small building in Palo Alto, which was not yet known as the heart of Silicon Valley. At the time, most computers — massive devices that stored data on punch cards or magnetic tapes — were housed in the offices of large companies and government laboratories. Woody's company could not afford to buy a computer, so scientists rented computing time on such a machine from their neighbors, often late at night when it was cheaper.
Panoramic's business was to "test ideas that we hoped would turn the world upside down."
According to Nels Winkless, a writer and consultant who was involved in several Panoramic projects and later co-founded Personal Computing magazine, "their job was to do what other people find too silly."
The inventions of some of the Panoramic researchers have become widely known. For example, Helen Chan Wolf, a pioneer in robot programming, worked on the Shakey the Robot. According to the Institute of Electrical and Electronics Engineers, it is "the world's first robot that embodies artificial intelligence."
Panoramic tried in vain to find funding. Woody did his best to present the character recognition technology, including presenting the invention to the Fair Life Insurance Society and McCall's magazine. But the contract was never signed.
Throughout its existence, Panoramic has had at least one reliable patron to keep it afloat, the Central Intelligence Agency.
If there were references to the CIA in Woody Bledsoe's papers, then most likely they were destroyed in 1995. But fragments of surviving material clearly indicate that Woody's company worked with CIA front companies for many years. Nels Winkless, who was friends with the Panoramic team, says the company was most likely created with agency funding. "No one ever told me directly about it," Winkless recalls, "but it was."
Panoramic Research Incorporated was one of 80 organizations working on the MK-Ultra project, according to the Free Access to Information Act (FIA) law inquiries site the Black Vault. This is the infamous CIA "mind control" program that used psychological torture without people's consent. Through the dummy research foundation the Medical Sciences Research Foundation, Panoramic was commissioned to undertake subprojects to study bacterial and fungal toxins and to "remotely control certain animal species."
David H. Price, an anthropologist at Saint Martin University, believed that Woody and his colleagues also received money from the Society for the Study of Human Ecology. On behalf of this society, the CIA provided grants to scientists whose work could improve the interrogation methods used by the agency, or act as a cover for such research.
But Panoramic's most significant research was provided by another fictitious company, the King-Hurley Research Group (King-Hurley). According to a series of lawsuits filed in the 1970s, the CIA used this research team to procure planes and helicopters for the secret air force agency known as Air America. For a time, King-Hurley also funded psychopharmacological research at Stanford.
In early 1963, King-Hurley was only accepting various presentation of ideas from Woody Bledsoe. He proposed to conduct "research to determine the feasibility of creating a simplified face recognition machine." Drawing on his work with Browning on the tuple method, Woody wanted to teach the system to recognize 10 faces. That is, he planned to use a database of 10 photographs of different people and find out if the machine can identify new photographs of each of them. “It will soon be possible to increase the number of people to thousands,” wrote Woody. Within a month, King-Hurley gave him permission to start work.
First experiments in the field of face recognition
Identifying ten people may seem like a rather modest goal today, but in 1963 it was incredibly ambitious. The leap from written character recognition to facial recognition has been enormous. If only because there was neither a standard method for digitizing photographs, nor an existing digital image base on which to rely. Modern researchers can train their algorithms on millions of free selfies, and Panoramic would have to build a database from scratch.
There was also a more serious problem: the three-dimensional faces of people, unlike two-dimensional signs, are not static. Images of the same person may differ in head rotation, light intensity and angle, as well as depending on age, hairstyle and mood - in one photo a person may appear carefree, in another - anxious.
By analogy with finding a common denominator in an extremely complex fraction, the team had to adjust for variability and order the images they were comparing.
And it would hardly be possible to say with confidence that their computers will cope with this task. One of the main machines was the CDC 1604 with 192 KB of RAM - about 21,000 times less than a typical modern smartphone.
From the beginning, Woody was fully aware of these complexities, so he took a divide and conquer approach: he broke the research into pieces and assigned them to different collaborators.
The work on digitizing the images took place as follows. The researcher shot black and white photographs of the project participants on 16mm film. Then he used the scanning device Browning had developed to transform each image into tens of thousands of data points. Each point had to have a light intensity value in the range from 0 (darkest) to 3 (lightest) - in a certain place in the image. There were too many points to be processed by the computer at a time, so the researcher wrote the NUBLOB program, which sliced the image into random samples and calculated a unique value for each - like those assigned using the tuple method.
Woody, Helen Chan Wolfe and another researcher worked on the head tilt. First, the scientists drew a series of numbered small crosses on the left side of the subject's face, from the top of the forehead to the chin. Then they made two portraits, in one of which the person was looking forward, and in the other - turned 45 degrees. After analyzing the location of the crosses in these two images, the data was extrapolated to a face image with a rotation of 15 or 30 degrees. A black-and-white picture of a marked face was loaded into a computer, and the output was an automatically rotating portrait - scary, pinpoint and surprisingly accurate.
The researchers' solutions were original, but not effective enough. Thirteen months after starting work, the Panoramic team admitted that they had failed to train the machine to recognize at least one face, let alone ten.
Hair growth, facial expressions, and signs of aging — this triple challenge represented “a colossal source of variability,” Woody wrote in his March 1964 progress report to King-Hurley. The set task "goes beyond the current state of the field of pattern recognition and modern computer technologies." In doing so, Woody recommended funding more research to try to find a "completely new approach" to solving the problem of facial recognition.
"Human-machine" approach to face recognition
Over the next year, Woody came to the conclusion that the most promising approach to automated face recognition is one that narrows the area down to the relationships between the main elements: eyes, ears, nose, eyebrows, lips.
The system he proposed was similar to that of the French criminologist Alphonse Bertillon, which he created in 1879. Bertillon described people based on 11 physical measurements, including the length of the left leg and the length from the elbow to the end of the middle toe. The idea was that if enough measurements were taken, each person would be unique. The method was laborious, but it worked: with the help of it in 1897, long before the widespread use of fingerprinting, the French gendarmes identified the serial killer Joseph Vas.
Throughout 1965 Panoramic tried to create a fully automated Bertillon face identification system. The team was trying to develop a program that could define noses, lips, and more using light and dark areas in a photograph. But they failed.
Then Woody and Wolfe took up what they called the "human-machine" approach to face recognition — a technique that incorporated a little human input into the equation.
Woody attracted his son Gregory and his friend to the project - they were given 122 photographs, which showed about 50 people. The guys took 22 measurements of each face, including the length of the ear and the width of the mouth. Wolfe then wrote a program to process the data.
. , : , .
Their next step, in late 1965, was to create a larger version of the same experiment to make "human" more efficient in their human-machine system. With King-Hurley's money, they bought the RAND tablet, a $ 18,000 device that looked like a flatbed image scanner but worked like an iPad. Using a stylus, the researcher drew on a tablet and at the output received a computer image of a relatively high resolution.
A new batch of photos was taken through the RAND tablet, emphasizing the key elements of the face with the stylus. This process, although complex, was much faster than before: data were entered for about 2,000 images, including at least two images of each face. About 40 images were processed per hour.
Even with this larger sample, Woody's team struggled to overcome the usual hurdles.
The problem with smiles, which “distort the face and radically change interfacial dimensions,” as well as aging, has not yet been resolved.
When trying to match Woody's 1945 photo with the 1965 photo, the system got confused. She did not see much similarity between a young man with a wide smile and thick dark hair and an older man with a somber expression and thinning hair.
Photo by Woody Bledsoe from a 1965 study. Photographer: Dan Winters
It was as if decades had created a different person - and in a sense it was. By this point, Woody was tired of looking for new contracts for Panoramic and found himself "in the ridiculous position of either too much work or not enough." He continually presented new ideas to his sponsors, some of which are today considered ethically questionable.
In March 1965 - 50 years before China began using facial matching to identify ethnic Uyghurs in Xinjiang province - Woody invited the Advanced Research Projects Agency (ARPA) of the US Department of Defense to support Panoramic in studying the use of traits. persons to determine the racial origin of a person. “There is a very large number of anthropological dimensions of people from all over the world who belong to different racial and ecological groups,” Woody wrote. "It is a vast and valuable data warehouse that has been collected with great difficulty and expense, but not used properly." Whether ARPA agreed to fund this project remains unknown.
Woody invested thousands of dollars in Panoramic from his own funds with no guarantee of their return. Meanwhile, his friends from the University of Texas at Austin persuaded him to get a job at the university, luring him with a stable salary. And in January 1966, Woody left Panoramic. The company closed shortly thereafter.
With the dream of creating a computer man, Woody moved with his family to Austin to devote himself to the study and teaching of automated reasoning. But his work on facial recognition technology did not end there.
Woody Bledsoe's most successful facial recognition experiment
In 1967, Woody took on a final assignment related to facial pattern recognition. The purpose of the experiment was to help law enforcement agencies quickly sift through the databases of arrested persons in search of matches.
As before, funding for the project appears to have come from the US government. A 1967 document, declassified by the CIA in 2005, mentions an "external contract" for a facial recognition system that would cut search times a hundredfold.
Woody's main project partner was Peter Hart, a research engineer in the Applied Physics Laboratory at Stanford Research Institute. (Now known as SRI International. The institute split from Stanford University in 1970 due to differences on campus over the institute's heavy reliance on military funding.)
Woody and Hart started with a database of about 800 images - two each of "400 Caucasian adult males." Those photographed differed in age and head turn. Using a RAND tablet, the scientists recorded 46 coordinates for each photograph, including five values for each ear, seven for the nose, and four for each eyebrow. Building on Woody's previous experience with normalizing image variation, a mathematical equation was used to "turn" the heads in frontal view. Then, to account for the difference in scale, each image was enlarged or reduced to a standard size, where the reference metric was the distance between the pupils.
The task of the system was to memorize one version of each person and use it to identify the other.
Woody and Hart offered the car one of two shortcuts. In the first, known as group match, the system divided the face into features - left eyebrow, right ear, and so on - and compared the relative distances between them.
Photographer: Dan Winters The
second approach was based on Bayesian decision theory, where the machine used 22 dimensions to make a general educated guess.
As a result, both programs coped with the task approximately equally well. And also turned out to be better than human rivals. When Woody and Hart asked three people to match subgroups of 100 individuals, even the fastest of them took six hours. The CDC 3800 completed a similar task in about three minutes, achieving a 100-fold reduction in time. Humans were better at handling head turns and poor photo quality, but the computer was "vastly superior" in determining age-related changes.
The researchers concluded that the machine "dominates" or "nearly dominates" the person. This was Woody's greatest success in his research on facial recognition.
It was also his last work on the topic, which was never published "in the interest of the state," which Woody and Hart greatly regretted.
In 1970, two years after the end of his collaboration with Hart, a robot technician named Michael Kassler told Woody that Leon Harmon of Bell Labs was planning a study on facial recognition. “I am outraged that this second kind of study will be published and will end up being the best man-machine system,” Woody replied. “I think with hard work Leon will be about 10 years behind us by 1975 year. ”Woody must have been disappointed when Harmon's research hit the cover of Scientific American a few years later - while his own more advanced work was kept in storerooms.
Using the Woody Bledsoe method in modern face recognition technology
In the decades that followed, Woody won awards for his contributions to automated reasoning. For a year, he served as President of the Association for the Development of Artificial Intelligence. But his work on facial recognition remained largely unrecognized and almost forgotten, while others were collecting laurels.
In 1973, Japanese computer scientist Takeo Kanade made a big leap in facial recognition technology.
Based on a database of 850 digitized photographs from the World's Fair in Sweet, Japan in 1970, Canada developed a program that could extract facial features - nose, mouth, and eyes - without human intervention. Canada has succeeded in fulfilling Woody's dream of excluding man from the man-machine system.
Woody has used his knowledge of facial recognition a couple of times over the years.
In 1982 he was brought in as an expert in a criminal case in California. An alleged member of the Mexican mafia was accused of committing a series of robberies in the Contra Costa County. The prosecutor had several pieces of evidence, including video surveillance footage of a long-haired man with a beard, sunglasses and a winter hat. But in the photographs, the accused appeared to be a clean-shaven man with short hair. Woody measured the robber's face, compared it with photographs of the accused, and found that the faces belonged to two different people due to the difference in the width of the noses. Despite the fact that the man still went to jail, he was acquitted of four counts, where Woody was a witness.
“In just the last 10 years, facial recognition technology has learned to deal with imperfections,” says Anil K. Jain, a software scientist at Michigan State University and co-editor of the Handbook of Face Recognition.
Almost all of the problems Woody faced have disappeared. Today there is an inexhaustible supply of digitized images. “Through social media, you can get as many face shots as you want,” says Jane. And thanks to advances in machine learning, memory, and processing power, computers learn to learn effectively. With a few simple rules, they can analyze huge amounts of data and create templates for almost anything from a human face to a bag of chips - no more measurements with a RAND tablet or the Bertillon method are required.
Even considering how far facial recognition has come since the mid-1960s, Woody Bledsoe identified many of the challenges that remain to be addressed in this area.
, - . , , , .
Although the programmer does not explicitly instruct modern deep learning systems to identify noses and eyebrows, Woody’s turn in that direction in 1965 set the direction of the industry for decades. “For the first 40 years it was feature extraction that dominated,” says Takeo Kanade, now a professor at the Carnegie Mellon Institute of Robotics.
Today, to some extent, they have returned to what resembles Woody's first attempts to "figure" a human face, when he used a variation of the tuple method to find patterns of similar features in a giant field of data points. As complex as modern face recognition systems are, Anil Jane says they simply compare pairs of images and assign them a similarity score.
But perhaps most importantly, Woody Bledsoe's work has set the ethical tone for facial recognition research - both relevant and problematic. Unlike other world-changing technologies whose catastrophic capabilities became apparent over the years - social media, YouTube, drones - the potential abuse of facial recognition technology has been evident almost from the moment it was born at Panoramic.
Many biases that can be attributed to the remnants of the time of Woody's research - the attraction to experiments of almost only white people, seemingly careless trust in government, the desire to use facial recognition to discriminate on racial grounds - all this is inherent in the modern field of face recognition.
In 2019 testing of Amazon's Rekognition software, 28 NFL players were mistakenly identified as criminals. A few days later, the American Civil Liberties Union filed a lawsuit with the US Department of Justice, the FBI and the Drug Enforcement Administration for information about their use of facial recognition technology from Amazon, Microsoft and other companies. A 2019 report from the National Institute of Standards and Technology, which tested the code of more than 50 facial recognition software developers, says white men are less likely to be mis-matched with criminals than other groups. In 2018, a couple of scientists came out with sharp criticism: "We believe that facial recognition technology is the most dangerous surveillance mechanism ever invented."
In the spring of 1993, due to a degenerative disease of ALS, Woody's speech deteriorated. But he continued to teach at the University of Texas until his speech became illegible. He continued his research in the field of automated reasoning - until he stopped holding the pen. Until the end, remaining a scientist, Woody took notes of his speech to track the development of the disease.
Woody Bledsoe died on October 4, 1995. The obituary did not mention his work in facial recognition. In the obituary photo, gray-haired Woody is staring straight into the camera, a broad smile on his face.
Commentary by Elena Gerasimova, Head of Analytics and Data Science at Netology
Woody Bledsoe's ideas were not commercially successful, perhaps because their time fell on one of the "artificial intelligence winters." There was little confidence in the technologies, there was not enough capacity to demonstrate impressive results, and the technologies for reconstructing the human brain were mainly engaged in by enthusiasts - academician Andrei Nikolaevich Kolmogorov, American mathematician George Tsibenko and others.
Nevertheless, thanks to this research, modern breakthroughs have become possible, which are based on powerful computing power, clouds, microchips.
In 1998, Yang LeCun perfected approaches to recognizing handwritten numbers on his LeNet - thanks to the evolution of computing power that was not available during the research of Woody Bledsoe.
Face recognition technology borders on more advanced face generation technology, which is used for, for example, creating deepfakes and generating faces of adults and children, as well as cats and dogs. It would seem that it is easier - to take a photo of a person and upload it, conditionally, to an electronic clothing catalog; or shoot a cute video with a baby and toys; or teach a neural network to create an image of a child in clothes, interior or with a toy that we plan to place in the catalog and thus demonstrate? The answer will be prompted by the amount of investment in companies developing technology for creating photorealistic images - in the United States alone in 2019, the total investment amounted to more than $ 500 million.
Generation of photorealistic images of people
If you are interested in working with data, neural networks, we offer you to get acquainted with our programs, training in which is based on a scientific base and research:
- Profession " Data Scientist ". Award " Quality Mark " in the nomination "Training of digital industry professionals" (2019).
- Deep Learning course . On the course landing page, you can see examples of projects within the course and see how the training is going.
And for those who are looking for approaches to the implementation of artificial intelligence, machine learning and "digital reboot", we suggest looking at the information on the course " Business transformation: the introduction of artificial intelligence ." Students create a strategy for the implementation of artificial intelligence and work out real business cases under the guidance of foreign and Russian experts.