Pramit Saha

Graduate student, University of British Columbia, Vancouver, Canada

Research interests

Deep Learning, Bayesian Inference, Medical Imaging, Computer Vision, Speech Synthesis

Selected recent projects

Jan2019 - Now 1. Investigating hand-to-speech motor control

The goal of this project is to investigate and develop a plausible underlying mechanism of how our human motor control system leverages the biomechanical and physical constraints as well as the auditory perceptual abilities to reduce the difficulty of the speech task.

Human speech production is one of the most complex processes within the human motor repertoire, that needs a precise coordination of over 100 muscles per utterance of word. In continuous speech, the brain has to deal with the challenging task of rapid and accurate coordination of a set of redundant and interacting articulators, which requires the multi-dimensional control of multiple articulators at a dauntingly high rate. The neuro-computational bases behind such control is still not well understood. In this work, we endeavor to address this issue by investigating the effect of perceptual feedback-based regulations and spatio-temporal constraints for control of vowel sounds in a hand gesture-to-speech mapping system. As a starting point, we developed a synthesizer which can be controlled by hand movement to produce continuous vowel sounds easily and intuitively. We also investigated glove based 2D control, 1D+1D control and mouse control to understand how different control paradigms vary the effort of hand-to-speech motor control. Furthermore, we put forth an information theoretic view of the aforementioned control and demonstrated how deep learning based mapping can be utilized to reduce the difficulty level of the task.


Peer-reviewed Publications:

[1] Pramit Saha and Sidney Fels. “Your Hands Can Talk : Perceptually-Aware Mapping of Hand Gesture Trajectories to Vowel Sequences” [under review]

[2] Yadong Liu* , Pramit Saha*, Arian Shamei, Bryan Gick and Sidney Fels. “Deep learning based continuous vowel space mapping from hand gestures”, Acoustics Week in Canada 2019

[3] Yadong Liu*, Pramit Saha*, Arian Shamei, Bryan Gick and Sidney Fels. “Mapping a Continuous Vowel Space to Hand Gestures.” Canadian Acoustics 48.1 (2020).

(* indicates equal contribution)

April2018 - Now

2. Mapping articulatory and acoustic domain

The goal of this project is to find appropriate mappings between articulatory geometric configurations of the vocal tract and the acoustic properties of the resultant speech sound.

Thousands of individuals need surgical removal of their larynx due to critical diseases every year and therefore, require an alternative form of communication to articulate speech sounds after the loss of their voice box. This project addresses the articulatory-to-acoustic mapping problem based on synthetic images (Pink Trombone), ultrasound (US) tongue images and MRI vocal tract images for the development of a silent-speech interface (SSI) that can provide them with an assistance in their daily interactions. We employed deep learning based approaches to develop MRI based speech recognition and US based speech sythesis systems. Further, in order to find a joint latent representation between the articulatory and acoustic domain for vowel sounds, we utilized a convolutional autoencoder architecture and normalizing flow-based model to allow both forward and inverse mapping between mid-sagittal vocal tract geometry (of a two degrees-of-freedom articulatory synthesizer with 1D acoustic wave model) and Mel-spectrogram representation (of the synthesized speech sounds).


Peer-reviewed Publications:

[1] Pramit Saha and Sidney Fels. “Learning Joint Articulatory-Acoustic Representations with Normalizing Flows.” arXiv preprint arXiv:2005.09463 (2020). [Accepted for publication in Interspeech 2020]

[2] Pramit Saha, Yadong Liu, Bryan Gick, and Sidney Fels. “Ultra2Speech - A Deep Learning Framework for Formant Frequency Estimation and Tracking from Ultrasound Tongue Images” [ MICCAI Society Young Scientist Award 2020] CODES AND SAMPLES

[3] Pramit Saha, Praneeth Srungarapu, and Sidney Fels. “Towards Automatic Speech Identification from Vocal Tract Shape Dynamics in Real-time MRI.” Proc. Interspeech 2018 (2018): 1249-1253.

Apr2018 - Apr2019 3. Recognizing imagined speech

The goal of the project is to detect speech tokens from speech imagery brain signals (EEG).

Speech imagery is about representing speech in terms of unspoken sounds inside the human brain that doesn’t involve overt vocalization or articulatory movements. Our project reveals the existence of some sort of brain footprint for articulatory movements underlying related speech token imagery. We build upon the idea that active thought process underlying covert speech does have some relevant features corresponding to the intended activity of different parts of the vocal tract, even though a person is not vocalizing. We further demonstrate that it is possible to detect the imagined words by understanding the intended involvement of vocal tract and vocal fold, which is internally encoded in the brain signals. Our hierarchical deep neural network architecture is able to capture some information about the signals that the brain implicitly sends to the speech articulators like tongue, vocal fold, etc even in the absence of a need for vocal communication.


Peer-reviewed Publications:

[1] Pramit Saha, Muhammad Abdul-Mageed, and Sidney Fels. “SPEAK YOUR MIND! Towards Imagined Speech Recognition with Hierarchical Deep Learning.” Proc. Interspeech 2019 (2019): 141-145.

[2] Pramit Saha, Sidney Fels, and Muhammad Abdul-Mageed. “Deep learning the eeg manifold for phonological categorization from active thoughts.” ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019.

[3] Pramit Saha and Sidney Fels. “Hierarchical deep feature learning for decoding imagined speech from eeg.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.

Jan2018 - Dec2018

4. Developing silent speech interfaces

This project explores and evaluates the appropriate input and mapping methods to design a controllable silent speech synthesis engine using either our vocal tract or hands. It has two components:

(i) Ultrasound (US) based SSI: We take a step towards developing a controllable interface via precise extraction of the upper oral cavity from ultrasound imaging modality, thereby, enabling speakers to drive an articulatory speech synthesizer directly by their tongue movements without the necessity of vocalization. At the core of this interface is the extraction of mid-sagittal cross-sectional area functions between tongue and palate using the US, which drives a computationally affordable FDTD-based 2D acoustic wave solver for precise simulation of acoustic wave propagation. The study motivates an investigation into the possible ways of augmenting US-based area functions to generate rich VT geometrical information that can potentially lead to the development of a real-time silent-speech interface.

(ii) Hand control based SSI: We introduce Sound stream: a low-cost, tangible and ambidextrous controller which drives a dynamic muscle-based model of the human vocal tract for articulatory speech synthesis. The controller facilitates the usage of multidimensional inputs from both hands which are mapped to the tongue muscles in a biomechanical modeling toolkit Artisynth using a microcontroller. The tongue kinematics is then mapped to a JASS based sound synthesis engine. As a demonstration, the user learns to interact and control a mid-sagittal view of the tongue structure in Artisynth through a set of sensors using both hands to synthesise continuous vocal sounds.


Peer-reviewed Publications:

[1] Yadong Liu, Pramit Saha, and Bryan Gick. “Visual Feedback and Self-monitoring in Speech Learning via Hand Movement” [The 179th Meeting of the Acoustical Society of America, 2020]

[2] Pramit Saha, Debasish Ray Mohapatra, S. V. Praneeth, and Sidney Fels. “Sound-Stream II: Towards Real-Time Gesture-Controlled Articulatory Sound Synthesis.” Canadian Acoustics 46.4 (2018): 58-59.

[3] Pramit Saha, Debasish R. Mohapatra, Venkata Praneeth Srungarapu, and Sidney Fels. “SOUND STREAM: Towards vocal sound synthesis via dual-handed simultaneous control of articulatory parameters.” The Journal of the Acoustical Society of America 144, no. 3 (2018).

Education

Sep2017 - Now M.A.Sc (Master of Applied Science), Electrical and Computer Engineering, University of British Columbia

Supervised By: Sidney Fels

Project: Mapping articulatory-to-acoustic domain and investigating speech motor control

Jul2012 - Jun2016 B.E.(Bachelor of Engineering) Honours (Rank: 5 out of 100 students) in Electrical Engineering, Jadavpur University, Kolkata, India

Advisor: Amitava Chatterjee

Project: Three-dimensional terrain image map generation for navigation of a quadruped robot in an uneven terrain.

Professional Experience

Sep2017 - Now MITACS Globalink Graduate Fellow and Graduate Research Assistant, HCT lab, Vancouver, Canada

May2018 - Now Treasurer and Executive Committee Member, UBC ECE Graduate Student Association (ECEGSA)

Jun2020 - Now Graduate Academic Assistant, Electrical and Computer Engineering Department (ECE), University of British Columbia (UBC), Vancouver, Canada

Jun2016 - Sep2016 MITACS Globalink Research Intern, Faculty of Medicine and Dentistry, Department of Radiology and Diagnostic Imaging, University of Alberta, Edmonton

May2015 - Jul2015 Summer Research Intern, Department of Electrical Engineering, Indian Institute of Science (IISc Bangalore)

Dec2014 - Jan2015 Winter Research Intern, Electronics and Electrical Communication Engineering Department, Indian Institute of Technology (IIT Kgp)

Teaching Experience

Jan2018 - Now Teaching Assistant, University of British Columbia, Vancouver, Canada

Responsibilities: Preparing course materials, conducting tutorial sessions, managing student projects, lab and exam grading

Reviewing Activities

Computing Skills

Programming Language: Python (including deep learning frameworks: PyTorch, Keras, TensorFlow), MATLAB, C/C++

Awards and Recognitions

2020

MICCAI 2020 Young Scientist Award awarded by the MICCAI Society

2020

MICCAI 2020 Student Travel Award awarded by the MICCAI Society

2019

ISCA Travel Grant awarded by the International Speech and Communication Association (ISCA) - 650 Euro

2019

AAAI Student Scholarship awarded by Association for Advancement of Artificial Intelligence (AAAI) - 400 Dollars

2018-2019

Faculty of Applied Science Graduate Award awarded for outstanding academic and research achievements by the Department of Electrical and Computer Engineering (ECE) of University of British Columbia (UBC) - 8,000 Canadian dollars

2017-Now

International Tuition Award awarded by the University of British Columbia - 9,600 Canadian Dollars

2017-2018

MITACS Globalink Graduate Fellowship Award awarded by MITACS - 15,000 Canadian Dollars

2016

MITACS Globalink Research Internship (GRI) Award awarded by MITACS - 6,800 Canadian Dollars

2015

Best Paper Award awarded by IEEE INDICON 2015

2014

Best Student Paper Award awarded by IEEE CALCON 2014

2014

Best White Paper Award 2014 awarded by Schneider Electric Pvt. Ltd.

2012-2016

Ministry of Human Resource Development Scholarship awarded for securing 27th rank out of 7,00,000 candidates (approx.) in the 12th standard Board Examinations, by Government of India - 50,000 INR

Publications

To see the complete list of my publications, please visit google scholar

Invited Talk

  1. UBC Language Sciences Talk: TEACHING, MAPPING, AND UNDERSTANDING THE SOUNDS OF LANGUAGE

Media / coverage

  1. What if you could hear new music by Satchmo? UBC researchers and speech from brain signals

  2. UBC Award Recipient Interview

  3. UBC ECE Graduate Student Achievement

Community and Volunteer Activities

2018-Now

Treasurer and Executive Committee Member, ECE Graduate Student Association (ECEGSA), University of British Columbia

As the treasurer and executive committee member of ECEGSA, I am in charge of organizing and maintaining funds for society of more than 300 graduate students in the department. I participate in organizing the academic seminars and tutorials as well as internal and external social events. I prepare budgets for each event, organize meetings with the other members of the committee, communicate with the bank and the departmental financial authorities, withdraw from and deposit money into the bank, audit the expenditures monthly, etc. Besides, I participate in the decision making of different departmental student welfare policies taking by ECEGSA (Electrical and Computer Engineering Graduate Student Association).

2017-2020

Lab Website Maintainer, Human Computer Technologies (HCT) Lab, University of British Columbia

I am volunteering to maintain the lab website and keeping it up-to-date. I am running tutorials to help the lab members understand how to create their profiles, include their publications and other details in their profiles, etc.

2012-2016

Class Representative and Member of Anti-Ragging Committee, Electrical Engineering Department, Jadavpur University

I acted as the Class Representative of Department of Electrical Engineering, Jadavpur University in my Undergraduate days, for four years. I was responsible for printing and distributing study materials and notes handed over by the course instructors to me, discussing with the students and deciding examination dates and communicating to the department examination committee. Besides, I was in charge of maintaining general discipline and academic environment in the class. Furthermore, I was the Student Member of Anti-ragging Committee of Jadavpur University (2012-2016), where I used to meet the other members and discuss about current ragging and disciplinary scenarios in the campus, reporting about unwanted incidents in the department, propose solutions for improving the anti-ragging policies, helping students with counselling, etc.

Students mentored

Himanshu Goyal, Third-year Computer Science student, UBC

Irene Wang, Third-year Computer Engineering student, UBC

Bobby Smith, Bachelor of Science (Biochemistry), UBC

Personal Home Page

HCT UBC Lab Profile

UBC Language Science Profile

UBC Meet our students Profile

UBC ECEGSA Treasurer Profile