How to build a sign-to-speech converter

Sign language is a system of communication using visual gestures. However, not everyone understands sign language. So, in this project, we’ll build a device that converts hand gestures into speech.

What’s required: flex sensors, a DF Mini Player, and a microcontroller that has an analog input and supports serial communication via a universal asynchronous receiver/transmitter (UART) — we use ESP32.

ESP32 offers multiple analog input channels, which nicely accommodate enough flex sensors to interface with a person’s hands and fingers (to relay the sign-language gestures). However, for this project, we’ll use a single flex sensor, which is enough to gesture for food, water, or medicine.

Adding more flex sensors to your device is possible, including enough to fully convert American sign language into audible speech.

Components required

ESP32 x1
DF Mini Player x1
MicroSD card x1
Speaker x1
Flex sensor x1
330Ω resistor x1
Connecting wires
Breadboard or Perfboard

Circuit connections
DOIT ESP32 DevKit Version 1 is used for this project. The pin diagram of this ESP board is below.

The DF Mini Player is used to play the audible sentences converted from sign language. It’s a small playback music system that plays audio files from a microSD card. The supported file formats include MP3, WAV, and WMA. The module has a microSD card slot and built-in amplifier and a 3.5mm audio jack for interfacing speakers or headphones. It features UART and USB interfaces to communicate with a microcontroller.

Here’s the DF Mini Player’s pin diagram:

The module has the following pin configurations:

The connections:

To connect the DF Mini Player with ESP32, connect the Mini Player’s VCC and GND pins with ESP32’s 3V3 and ground pins, respectively.
Next, connect the DF Mini Player’s RX and TX pins with ESP32’s GPIO26 and GPIO27 pins.
Then, connect a speaker (<3W) with the DF Mini Player’s SPK1 and SPK2 pins.

To play the DF Mini Player, audio files must be loaded to a MicroSD card. We’ll play three audio instructions from the device. These instructions can be converted from text to speech and downloaded as MP3 files from ttsmp3.com.

The audio instructions are attached as MP3 files below.

Upload these files to a MicroSD card and insert the card in the DF Mini Player’s slot.

A flex sensor is used to detect hand gestures (sign language). This sensor is connected to a voltage divider circuit with analog input at the GPIO36 pin.

The circuit diagram for the device:

The flex sensor can be secured to a glove or attached to a person’s fingers with a tape, so it bends with one’s finger as they use gestures to communicate with sign language.

Libraries required
To build this project, you’ll need to install two libraries in Arduino IDE – espSoftwareSerial and DFRobotDFMiniPlayer. The espSoftwareSerial is required for using the software serial on ESP32. The DFRobotDFMiniPlayer is used to work with the DF Mini Player.

Arduino sketch

How it works
The flex sensor detects one’s hand gestures as they communicate via sign language. When the sensor is bent, its resistance changes, so the voltage output by the voltage divider circuit also changes.

The flex sensor’s analog output is detected by ESP32 and used to determine the instruction/command. The DF Mini Player plays the MP3 files from the MicroSD card based on the instructions communicated via ESP32 over the UART.

The code
The sketch begins by importing SoftwareSerial.h and DFRobotDFPlayerMini.h. The variables are declared for the DF Mini Player’s RX and TX pin connections, and an object of the software serial is instantiated.

Next, an object of the DF Mini Player is instantiated, and the variables from the analog input’s connection to the flex sensor are declared. A variable to store the analog value is defined and initialized at zero.

In the setup() function, the baud rate for both the serial and software serial communication is set to 9600 bps. The DF Mini Player is initialized, and its volume is set to 20.

In the loop() function, ESP32 reads the analog input from the flex sensor. According to the value of the analog input, the MP3 file for the detected gesture is played by calling the player.play() method.

Results