Search for resonances decaying to an anomalous jet and a Higgs boson in proton–proton collisions at $$\sqrt{s}=13\,\text {Te}\hspace{-.08em}\text {V} $$

This paper presents a search for new physics through the process where a massive particle, X, decays into a Higgs boson and a second particle, Y. The Higgs boson subsequently decays into a bottom quark–antiquark pair, which is reconstructed as a single large-radius jet. The decay products of Yare also assumed to produce a single large-radius jet. The identification of the Yparticle is enhanced by computing the anomaly score of its candidate jet using an autoencoder, which measures deviations from typical quark- or gluon-induced jets. This allows a simultaneous search for multiple Ydecay scenarios within a single analysis. In the main benchmark process, Yis a scalar particle that decays into a Wboson pair. Two other scalar Ydecay processes are also considered as benchmarks: decays to a light quark–antiquark pair, and decays to a top quark–antiquark pair. A fourth benchmark process considers Yas a hadronically decaying top quark, arising from the decay of a vector-like quark into a top quark and a Higgs boson. Data recorded by the CMS experiment at a center-of-mass energy of 13TeV in 2016–2018, corresponding to an integrated luminosity of 138fb-1, are analyzed. The search covers Xmasses between 1.4 and 3.0TeV and Ymasses between 90 and 400GeV, with all simulated signals produced in the narrow-width approximation. No significant excess above the standard model background expectation is observed.