Evaluting Azure Video Indexer
Background
A pillar of what I am trying to do with the farm is to demonstrate that reinforcement learning can operate as a framework to drive positive societal change if people share data in a specific format. From a RL perspective, all an algorithm needs are state, action, state and individual farms would provide the reward based on the company mission.
This weekend at a cafe expo I learned about B Corporations which fits well with RL since it provides states, actions and rewards for the societal impact of a business.
In this article, I tried out Azure Video Indexer on a video that is similar to what farm video might contain. In an ideal scenario, Azure would identify key moments that correspond to states and actions that predict future rewards. For instance in this case the state could be thick grass on a terrace terrace, the action is plow and a new state might be 80% less green. Chain enough actions and states and eventually we get coffee, rice or vegetables which can be sold for a reward. RL will assign rewards to each action in the sequence.
If different actions are taken in different places, the system can learn actions that correlate with rewards. At that point randomized experiments could be ran to identify causation. If a coffee franchise were created that included a technology package, every action and state on a farm could be used to learn how actions impact our values.
If the Azure video stops working here is the Google version.
Ideal images to produce
0s: Shows the before state of the field
12s: Shows the action of plowing
24s: the 1st pass is completed
29s: shows the carabao action of eating
1:29: carabao starts plowing
1:53 carabao is eating and plowing
Azure Video Indexer Results
The video was created 36min ago but there is still a spinning circle on the video. The scenes and shots are a hierarchical way of looking at data.
This might be helpful on more complex tasks like construction where ROI could be calculated for purchasing different tools.
Scene 1
17s represents when the carabao passed the camera
38 is when the carabao passed Dhodie on the left
109 showed a different carabao
Scene 2
Covers when the camera transitioned between locations.
Scene 3
1:19 Shows waiting while eating
1:46 Transition from side of Dhodie to the back
2:16 Camera zooms in
Summary
Without domain specific guidance, a reasonable way to define key moments is when predictions of what is an image change quickly. Transitioning from the side to the back of a person might mean different datasets and models become activated.
Off-the-shelf, Azure Video Indexer doesn’t capture farm operations but could be helpful to organize training materials, or understand items such as construction.
Open source security software such as ispy seem better geared towards what I am trying to do.
