patents

2023

2023

1. US, EU

   Method and system for visual context aware automatic speech recognition

   Chayan Sarkar, Pradip Pramanick, and Ruchira Singh

   2023

   US Patent App. US20240038224A1, European Patent App. EU4325482A1

   Abs HTML

   Accuracy of transcript is of foremost importance in Automatic Speech Recognition (ASR). State of the art system mostly rely on spelling correction based contextual improvement in ASR, which is generally a static vocabulary based biasing approach. Embodiments of the present disclosure provide a method and system for visual context aware ASR. The method provides biasing using shallow fusion biasing approach with a modified beam search decoding technique, which introduces a non-greedy pruning strategy to allow biasing at the sub-word level. The biasing algorithm brings in the visual context of the robot to the speech recognizer based on a dynamic biasing vocabulary, improving the transcription accuracy. The dynamic biasing vocabulary, comprising objects in a current environment accompanied by their self and relational attributes, is generated using a bias prediction network that explicitly adds label to objects, which are detected and captioned via a state of the art dense image captioning network.

2. US, EU

   Methods and systems for disambiguation of referred objects for embodied agents

   Chayan Sarkar, Pradip Pramanick, Brojeshwar Bhowmick, Ruddra Dev Roychoudhury, and Paul Sayan

   2023

   US Patent App. US20240013538A1, European Patent App. EU4303740A1

   Abs HTML

   This disclosure addresses the unresolved problems of tackling object disambiguation task for an embodied agent. The embodiments of present disclosure provide a method and system for disambiguation of referred objects for embodied agents. With a phrase-to-graph network disclosed in the system of the present disclosure, any natural language object description indicating the object disambiguation task can be converted into a semantic graph representation. This not only provides a formal representation of the referred object and object instances but also helps to find an ambiguity in disambiguating the referred object using a real-time multi-view aggregation algorithm. The real-time multi-view aggregation algorithm processes multiple observations from an environment and finds the unique instances of the referred object. The method of the present disclosure demonstrates significant improvement in qualifying ambiguity detection with accurate, context-specific information so that it is sufficient for a user to come up with a reply towards disambiguation.

2022

2022

1. US

   Telepresence robots having cognitive navigation capability

   Snehasis Banerjee, Pradip Pramanick, Chayan Sarkar, Abhijan Bhattacharyya, SAU Ashis, Kritika Anand, Ruddra Dev Roychoudhury, and Brojeshwar Bhowmick

   2022

   US Patent App. US20230213941A1

   Abs HTML

   The embodiments of present disclosure herein address unresolved problem of cognitive navigation strategies for a telepresence robotic system. This includes giving instruction remotely over network to go to a point in an indoor space, to go an area, to go to an object. Also, human robot interaction to give and understand interaction is not integrated in a common telepresence framework. The embodiments herein provide a telepresence robotic system empowered with a smart navigation which is based on in situ intelligent visual semantic mapping of the live scene captured by a robot. It further presents an edge-centric software architecture of a teledrive comprising a speech recognition based HRI, a navigation module and a real-time WebRTC based communication framework that holds the entire telepresence robotic system together. Additionally, the disclosure provides a robot independent API calls via device driver ROS, making the offering hardware independent and capable of running in any robot.

2. US, EU

   System and method for ontology guided indoor scene understanding for cognitive robotic tasks

   Snehasis Banerjee, Balamuralidhar Purushothaman, Pradip Pramanick, and Chayan Sarkar

   2022

   US Patent App. US20230162494A1, European Patent EU4170449B1 (Granted)

   Abs HTML

   Existing cognitive robotic applications follow a practice of building specific applications for specific use cases. However, the knowledge of the world and the semantics are common for a robot for multiple tasks. In this disclosure, to enable usage of knowledge across multiple scenarios, a method and system for ontology guided indoor scene understanding for cognitive robotic tasks is described where in scenes are processed based on techniques filtered based on querying ontology with relevant objects in perceived scene to generate a semantically rich scene graph. Herein, an initially manually created ontology is updated and refined in online fashion using external knowledge-base, human robot interaction and perceived information. This knowledge helps in semantic navigation, aids in speech, and text based human robot interactions.

2021

2021

1. US, EU

   Methods and systems for enabling human-robot interaction to resolve task ambiguity

   Chayan Sarkar, Pradip Pramanick, Snehasis Banerjee, and Brojeshwar Bhowmick

   2021

   US Patent US11501777B2 (Granted), European Patent EU3995266C0 (Granted)

   Abs HTML

   The disclosure herein relates to methods and systems for enabling human-robot interaction (HRI) to resolve task ambiguity. Conventional techniques that initiates continuous dialogue with the human to ask a suitable question based on the observed scene until resolving the ambiguity are limited. The present disclosure use the concept of Talk-to-Resolve (TTR) which initiates a continuous dialogue with the user based on visual uncertainty analysis and by asking a suitable question that convey the veracity of the problem to the user and seek guidance until all the ambiguities are resolved. The suitable question is formulated based on the scene understanding and the argument spans present in the natural language instruction. The present disclosure asks questions in a natural way that not only ensures that the user can understand the type of confusion, the robot is facing; but also ensures minimal and relevant questioning to resolve the ambiguities.

2020

2020

1. US, EU

   Knowledge partitioning for task execution by conversational tele-presence robots in a geographically separated environment

   Chayan Sarkar, Snehasis Banerjee, Pradip Pramanick, Hrishav Bakul Barua, Soumyadip Maity, Dipanjan Das, Brojeshwar Bhowmick, Ashis Sau, Abhijan Bhattacharyya, Arpan Pal, and  others

   2020

   US Patent US11597080B2 (Granted), European Patent App. EU3881988A1

   Abs HTML

   Conventional tele-presence robots have their own limitations with respect to task execution, information processing and management. Embodiments of the present disclosure provide a tele-presence robot (TPR) that communicates with a master device associated with a user via an edge device for task execution wherein control command from the master device is parsed for determining instructions set and task type for execution. Based on this determination, the TPR queries for information across storage devices until a response is obtained enough to execute task. The task upon execution is validated with the master device and user. Knowledge acquired, during querying, task execution and validation of the executed task, is dynamically partitioned by the TPR across storage devices namely, on-board memory of the tele-present robot, an edge device, a cloud and a web interface respectively depending upon the task type, operating environment of the tele-presence robot, and other performance affecting parameters.

2. US, EU

   Robotic task planning for complex task instructions in natural language

   Pradip Pramanick, Hrishav Bakul Barua, and Chayan Sarkar

   2020

   US Patent US11487577B2 (Granted), European Patent App. EU3859587A1

   Abs HTML

   This disclosure provides systems and methods for robotic task planning when a complex task instruction is provided in natural language. Conventionally robotic task planning relies on a single task or multiple independent or serialized tasks in the task instruction. Alternatively, constraints on space of linguistic variations, ambiguity and complexity of the language may be imposed. In the present disclosure, firstly dependencies between multiple tasks are identified. The tasks are then ordered such that a dependent task is always scheduled for planning after a task it is dependent upon. Moreover, repeated tasks are masked. Thus, resolving task dependencies and ordering dependencies, a complex instruction with multiple interdependent tasks in natural language facilitates generation of a viable task execution plan. Systems and methods of the present disclosure finds application in human-robot interactions.

3. US, EU

   System and method for enabling robot to perceive and detect socially interacting groups

   Hrishav Bakul Barua, Pradip Pramanick, and Chayan Sarkar

   2020

   US Patent US11354531B2 (Granted), European Patent EU3929803B1 (Granted)

   Abs HTML

   This disclosure relates to system and method for enabling a robot to perceive and detect socially interacting groups. Various known systems have limited accuracy due to prevalent rule-driven methods. In case of few data-driven learning methods, they lack datasets with varied conditions of light, occlusion, and backgrounds. The disclosed method and system detect the formation of a social group of people, or, f-formation in real-time in a given scene. The system also detects outliers in the process, i.e., people who are visible but not part of the interacting group. This plays a key role in correct f-formation detection in a real-life crowded environment. Additionally, when a collocated robot plans to join the group it has to detect a pose for itself along with detecting the formation. Thus, the system provides the approach angle for the robot, which can help it to determine the final pose in a socially acceptable manner.

4. US, EU

   Conversational systems and methods for robotic task identification using natural language

   Pradip Pramanick, Chayan Sarkar, Balamuralidhar Purushothaman, Ajay Kattepur, Indrajit Bhattacharya, and Arpan Pal

   2020

   US Patent US11328726B2 (Granted), European Patent App. EP3804915A1

   Abs HTML

   This disclosure relates generally to human-robot interaction (HRI) to enable a robot to execute tasks that are conveyed in a natural language. The state-of-the-art is unable to capture human intent, implicit assumptions and ambiguities present in the natural language to enable effective robotic task identification. The present disclosure provides accurate task identification using classifiers trained to understand linguistic and semantic variations. A mixed-initiative dialogue is employed to resolve ambiguities and address the dynamic nature of a typical conversation. In accordance with the present disclosure, the dialogues are minimal and directed to the goal to ensure human experience is not degraded. The method of the present disclosure is also implemented in a context sensitive manner to make the task identification effective.

2019

2019

1. US, EU, JP

   Method and system for online non-intrusive fatigue-state detection in a robotic co-working environment

   Chayan Sarkar, and Pradip Pramanick

   2019

   US Patent US10588579B2 (Granted), European Patent EP3593959B1 (Granted), Japan JP6854855B2 (Granted)

   Abs HTML

   A method and a robotic system for online localized fatigue-state detection of a subject in a co-working environment using a non-intrusive approach is disclosed. A force sensor, mounted on the robotic system is capable of capturing effective force applied by local muscles of the subject co-working with the robotic system, providing a non-intrusive sensing. The captured force is analyzed on-line by the robotic system 102 to detect current fatigue state of the subject and proactively predict the future state of the subject. Thus, enables alerting the subject before time avoiding any possible accident.