My first multivariate PULSAR experiment was fairly direct: join the channels end to end and pass the result to the original classifier. It gave me something to run and compare. It also left a question I still haven’t fully answered: how much information do I lose by treating several channels as one long signal?

I’m discussing this work in weekly meetings with Nestor. I want to use these notes to keep track of what I tested and what I can actually say about the results, rather than just showing the highest accuracy I found.

The first benchmark

On 13 October I completed the concatenated baseline on 19 UEA datasets, using the original train/test splits and seed 42. The equal-dataset mean accuracy was 77.86%. Handwriting scored 46.35% and EthanolConcentration scored 55.51%. That average was a record of one run per dataset, not evidence that I had improved PULSAR in general.

Keeping the channels separate

I then ran the PULSAR feature extractor on each channel separately and combined the features. After that, I added descriptions of how pairs of channels move together over short windows. I included small time shifts because a pattern might appear slightly later in one channel than another. The input stays in its original three-dimensional shape: cases, channels and time.

There are many possible channel pairs, windows and shifts, so I couldn’t keep everything. The relationship selector checks training folds, considers estimated calculation cost, and keeps a limited set of useful groups. It refits the supervised PULSAR extraction inside those folds. The final predictions still use the existing Ridge and Extra Trees classifiers.

The control changed my interpretation

I tested four datasets with seeds 42 to 46, keeping the original splits fixed. The relationship model averaged 46.75% on Handwriting, compared with 41.84% for the channel-only control. But on SelfRegulationSCP2, EthanolConcentration and HandMovementDirection, the channel-only version did better. EthanolConcentration reached 78.17% with relationships, a large increase over the first concatenated run, yet the newer channel-only control reached 79.24%. I can’t credit that increase to the relationships.

I also tried changing the selector’s validation loss from squared error to a classification-margin loss. Handwriting rose to 47.65%, but the other results were mixed. That experiment is separate from the probability blend below. I haven’t combined the two changes or established that either one should replace the original selector.

The latest run: choose how much to blend

The latest completed experiment finished on 16 October. It blends probabilities from the channel-only model and the original relationship model. For each dataset and seed, three outer training folds produce held-out predictions, with the whole candidate pipeline refitted inside each fold. I choose a relationship-model weight from 0, 0.25, 0.5, 0.75 and 1 using those training predictions. The test labels don’t choose the weight.

Across five seeds, the blend averaged 57.67% on SelfRegulationSCP2, 78.94% on EthanolConcentration, 46.61% on Handwriting and 55.68% on HandMovementDirection. Compared with the original relationship model, that is an increase of 1.44 and 0.76 percentage points on the first two datasets, and decreases of 0.14 and 0.27 points on the other two. The channel-only control still has the higher mean on three of the four.

What I still need to check

These are five seeds on fixed splits, not five independent resamples. Handwriting also has two classes with only two training examples each. Those examples had to stay in training, so blend validation covered 146 of 150 cases and 24 of 26 classes. More broadly, I’ve already looked at these four test sets while developing the method. I need untouched datasets or resamples before treating an improvement as confirmation.

Next I want to try windows defined as a percentage of recording length, rather than the same number of samples for every dataset. That is still a proposed experiment. For now, the result I find most useful is the comparison with the channel-only control: extra relationships helped Handwriting, but I don’t yet know a reliable way to decide when they will help elsewhere.

Code and results

I’ve attached the Python extension, the blending wrapper, tests and CSV results to the PULSAR project page. The original PULSAR implementation is included with its source revision and licence. My work here is the multivariate adaptation and experiments, not the underlying PULSAR algorithm.