1. Sequential Decision Problems with Weak Feedback

Many variants of sequential decision problems that are considered in the literature depend upon the type of feedback and the amount of information they reveal about the associated rewards. Most of the prior work studied the cases where feedback from actions reveals rewards associated with the actions. However, in many areas like crowd-sourcing, medical diagnosis, and adaptive resource allocation, feedback from actions may be weak, i.e., may not reveal any information about rewards at all. Without any information about rewards, it is not possible to learn which action is optimal. Clearly, learning an optimal action is only feasible in such cases if the problem structure is such that an optimal action can be identified without explicitly knowing the rewards. Our goal is to study the class of problems where one can infer optimal action without explicitly knowing the rewards.

Specifically, we study the Unsupervised Sequential Selection (USS) problem (depicted in Fig. 1), where rewards/losses for selected actions are never revealed, but the problem structure is amenable to identify the optimal actions. We also introduce a novel setup named Censored Semi-Bandits (CSB), where the reward observed from an action depends on the amount of resources allocated to it. These setups find applications in various fields like poaching, healthcare, crowd-sourcing, security, police patrolling, stochastic network utility maximization (shown in Fig. 2) and so on. Our work also focuses on distributed learning in multi-player multi-armed bandits for identifying the best subset of actions. The setup is such that the reward is only observed for those actions which are played by only one player. We developed near-optimal algorithms with theoretical guarantees for these problems whose empirical performance is validated on different problem instances derived from synthetic and real datasets.

Figure 1

Figure 2

The following research papers came out of our work:

Publications:

1. Arun Verma, Manjesh K Hanawal, Csaba Szepesvári, and Venkatesh Saligrama; Online Algorithm for Unsupervised Sequential Selection with Contextual Information.
   In Advances in Neural Information Processing Systems (pp. 778–788). 2020.

2. Arun Verma, Manjesh K Hanawal, and N. Hemachandra; Thompson sampling for unsupervised sequential selection.
   In Asian Conference on Machine Learning (pp. 545-560). 2020. PMLR.

3. Arun Verma, Manjesh K Hanawal, and N. Hemachandra; Unsupervised Online Feature Selection for Cost-Sensitive Medical Diagnosis.
   In 2020 International Conference on COMmunication Systems \& NETworkS (COMSNETS) (pp. 1-6). 2020.

4. Arun Verma and Manjesh K Hanawal; Stochastic Network Utility Maximization with Unknown Utilities: Multi-Armed Bandits Approach.
   In IEEE INFOCOM 2020-IEEE Conference on Computer Communications (pp. 189-198). 2020.

5. Arun Verma, Manjesh K Hanawal, Arun Rajkumar, and Raman Sankaran; Censored Semi-Bandits: A Framework for Resource Allocation with Censored Feedback.
   In Advances in Neural Information Processing Systems (pp. 14526-14536). 2019.

6. Arun Verma, Manjesh K Hanawal, and Rahul Vaze; Distributed algorithms for efficient learning and coordination in ad hoc networks.
   In 2019 International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOPT) (pp. 1-8). 2019.

7. Arun Verma, Manjesh K Hanawal, Csaba Szepesvári, and Venkatesh Saligrama; Online Algorithm for Unsupervised Sensor Selection.
   In The 22nd International Conference on Artificial Intelligence and Statistics (pp. 3168-3176). 2019.

8. Arun Verma and Manjesh K Hanawal; Unsupervised cost sensitive predictions with side information.
   In Proceedings of the ACM India Joint International Conference on Data Science and Management of Data (pp. 364-367). 2018.

2. INTERNET ECONOMICS AND NET NEUTRALITY

An increase in Internet traffic has caused serious network congestions, especially in wireless networks, which are bandwidth constrained. To sustain such rapid traffic growth and enhance user experiences, ISPs need to upgrade their network infrastructures and expand capacities. This in turn, benefits Content Providers (CPs) and they generate more revenue through monetization of demand. The CPs, who are exploring various smart data pricing (SDP) schemes, to further improve their revenues, prefer that their content is treated on priority or allowed access at lower data charges by the ISPs for any monetary transaction to happen. Such bilateral arrangements between CPs and ISPs may constitute neutrality violations and can further skew the CP marketplace. Our goal is to study these net neutrality issues and their impact on the internet ecosystem, including service providers and end-users.

We study the impact of zero-rating of content by CP by modeling the interaction between CPs, ISPs, and end-users through the Stackelberg formulation, where sponsoring of the content (differential pricing) is allowed in the Internet and end-users select CPs based on the quality of service (QoS) and data access charges. We also study the revenue sharing model between CPs and ISPs through the principal-agent framework. CPs determine contracts, and ISPs strategically determine the infrastructure enhancement for each CP. We compare neutral versus non-neutral scenarios based on the differentiation of action by ISPs. We also study the collaborative setting through Nash Bargaining, which improves the utilities of all players. We propose a revenue-sharing mechanism through an intermediate regulatory regime that we call "soft-neutral" which may benefit all the players in the long run.
The following research papers came out of our work:

Publications:

1. Fehmina Malik, Manjesh K. Hanawal, Yezekael Hayel, and Jayakrishnan Nair; Revenue sharing on the Internet: A case for going soft on neutrality regulations. [link]
   Performance Evaluation 145: 102152.

2. Fehmina Malik, Manjesh K. Hanawal, and Yezekael Hayel; Zero-Rating of Content and Its Effect on the Quality of Service in the Internet. [link]
   IEEE/ACM Transactions on Networking 28, no. 6 (2020): 2671-2684.

3. Fehmina Malik, Manjesh K. Hanawal, Yezekael Hayel, and Jayakrishnan Nair; Revenue Sharing in a Soft-Neutral Regime. link
   In 2020 International Conference on COMmunication Systems & NETworkS (COMSNETS), pp. 900-902. IEEE, 2020.

4. Fehmina Malik, and Manjesh K. Hanawal; Revenue Sharing Between Service Providers in the Internet: A Moral Hazard Approach. link
   In 2019 11th International Conference on Communication Systems & Networks (COMSNETS), pp. 420-423. IEEE, 2019.

5. Manjesh K Hanawal, Fehmina Malik, and Yezekael Hayel; Differential pricing of traffic in the Internet. link
   In 2018 16th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt), pp. 1-8. IEEE, 2018.

3. Net Neutrality Violation Detection

The regulators and end-users need net-neutrality violation detection systems to ensure and ascertain non-discriminated Internet access by ISP as proposed by "Net-neutrality." We developed a measurement framework called "FairNet" to cater to this requirement. Currently, it supports traffic differentiation (TD) detection using the active proving method. The methodology consists of the QoS parameters' performance comparison after normalizing its variations due to Internet-borne effects and service-specific implementations. It compares similar services, e.g., video streaming, and uses the ``common server" with the same data delivery scheme per service type. The appropriate Server Name Identification in the TLS handshake ensures the authenticity of probing traffic. The TD detection engine employs algorithms based on QoS performance comparison and connectivity status monitoring. The framework, deployed over the live Internet, provides the "FairNet App" application on Android and iOS platforms to end-user testing net-neutrality violation detection.

FairNet measurement framework

Overview : Encrypted Server Name Identification

The Server Name Identification or SNI is a plaintext parameter in TLS handshake that uniquely identifies the service, enabling per service traffic classification. We developed a method to mask/encrypt the SNI that applies to HTTPS-based services. The methodology consists of modifying the existing TLS protocol without adding an extra method for retaining all security benefits of the existing protocol. It implements a modified handshake with two back-to-back handshakes, one with SNI absent and the second one with the intended SNI. The server always accepts the handshake request without SNI and responds with the default security certificate. We demonstrated the method using python based client-server system interacting over the live Internet.
The following research papers came out of our work:

Publications:

1. Vinod S. Khandkar, Manjesh K. Hanawal; Detection of Traffic Discrimination in the Internet.
   2020 International Conference on COMmunication Systems & NETworkS (COMSNETS).

2. Vinod S. Khandkar, Manjesh K. Hanawal; Challenges in Net Neutrality Violation Detection: A Case Study of Wehe Tool.
   2021 International Conference on COMmunication Systems & NETworkS (COMSNETS).

3. Vinod S. Khandkar, Manjesh K. Hanawal; Masking Host Identity on Internet: Encrypted TLS/SSL Handshake;
   arXiv preprint arXiv:2101.04556, 2021.

Patents:

"FairNet : Measurement setup for Detection Net neutrality Violations", Indian patent, application no. 202021048922.

"Masking Host Identity on Internet: Encrypted TLS/SSL Handshake", Indian patent, application no. 202021055538.

4. Machine Learning in Next Generation Wireless Networks

The next-generation communication networks promise to enable connectivity for all devices to talk to each other and exchange information, giving rise to concepts like the Internet of things (IoT). I intend to study IoT systems that address the challenges for improving robustness and energy efficiencies, where IoT devices harvest energy from wireless radio frequency (RF) signals that can significantly increase their lifespan. Recently, introducing artificial intelligence (AI) in IoT systems enables these networks to learn from the environment and use that information to make intelligent decisions without human intervention. The successful implementation of Machine learning (ML), one of the most effective AI tools, enhances cognitive radio networks and spectrum sharing performances, which motivated me to develop ML algorithms for emerging fields of wireless networks. Presently, my focus is on the improvement of the efficiency of power harvesting using RF signals, as extracting energy from far-field radio frequency (RF) signal transmissions is a promising approach that opens up a new avenue for the unified study of wireless energy transfer and information transmission as radio signals are capable of simultaneously carrying energy and data.

We focused on the learning and fairness issues in RFEH sensor networks (refer Fig. 1) that ensure every node gets energy and remains active for a longer duration since the lifespan of the sensor network depends on the amount of energy a node can harvest in the network, which further depends on their ambient environment and proximity to the energy source. We have also proposed the power selection strategy of Wirelessly powered communication networks (refer Fig. 2) where the source learns the optimal transmit power that maximizes the received sum rate of the transmitted codewords by the nodes per unit power transmitted, defined as Energy Efficiency, at the source. This strategy avoids the wastage of power at the source because the sum rate received at the source will not significantly increase if the source increases the transmit power after a certain transmit power level, as the amount of energy harvested at the nodes will saturate. We set both the issues as Multi-Armed Bandits problems and used suitable stochastic bandits algorithms to solve them.

Fig. 1: Energy Harvesting set up

Fig. 2: Energy Harvesting set up

The following are two peer-reviewed publications of our work that look into the issues on RFEH:

Journal Publications:

1. Debamita Ghosh, Manjesh K. Hanawal, and Nikola Zlatanov; Learning to Optimize Energy Efficiency in Energy Harvesting Wireless Sensor Networks.
   IEEE Wireless Communications Letters (2021).

Conferences:

1. Debamita Ghosh, Arun Verma, and Manjesh K. Hanawal; Learning and Fairness in Energy Harvesting: A Maximin Multi-Armed Bandits Approach.
   In 2020 International Conference on Signal Processing and Communications (SPCOM), pp. 1-5. IEEE, 2020.