I agree that our current practices of chemistry, process monitoring and process control creates barriers and beliefs that limit our application of on-line monitoring. We prefer methods with chemical reagents because there are a limited number of corrections required for a good measurement. In contrast, there is a belief or prejudice that light coupled with spectrographic methods is too "blackbox". The great majority of chemical information is detectable via the electromagnetic force. However, we as chemists limit our expectation to specific or characteristic chemical spectra. Yet, we know that spectra is a result of electron sharing being restricted to specific time-spaces therefore only specific quanta occur when the electron is promoted from those restricted spaces. Further, we use our knowledge of mixing of these restricted spaces to predict where a reaction occurs and what new quanta to expect. Yet, what we generally miss is that a free electron also interacts with light. Hence, information about the chemistry can bias spectral distributions at wavelengths which are not those of the characteristic chemical spectra and do it in specific ways.
For example, I once helped develop a florescence detection method for the probability of 1 E. coli per 100 ml of sample of river or lake water. We were sampling less than 3 ml. A large number of characterization patterns (source water samples) were required for a database but then a comparison program could identify the most similar sample in the database and thereby accurately predict the most probable number of E. coli with the above limit of accuracy.
I expect that this method would put most of us out of our comfort zone.
