Keir Starmer was increasingly confident that he had seen off the immediate threat to his job on Tuesday after a challenge from Wes Streeting failed to materialise despite several of the health secretary’s allies quitting the government.

Downing Street insiders suggested that the health secretary did not yet have the required support from the 81 MPs he needed to formally launch a leadership bid after Starmer issued a ‘put up or shut up’ ultimatum to his cabinet.

Streeting was due to hold talks with Starmer on Wednesday, at which he was expected to talk candidly about his concerns, with No 10 insiders suggesting he was climbing down from intense speculation that he was on the brink of running.

“After all that, it’s looking like Wes may not have the numbers after all,” one loyalist cabinet minister told the Guardian. “I’m pleased and furious at the same time. The best thing for him now is to come out with some dignity and end the drama.”

Starmer’s allies also believe he has seen off a threat from the Greater Manchester mayor, Andy Burnham, for the time being, with one saying: “Andy’s supporters keep saying he’s got a seat. But where is it? It’s not real unless he has one.”

However, the prime minister’s fragile authority has been further weakened by the resignation of four ministers – three of them close allies of Streeting – in what appeared to be an orchestrated move. More than 90 Labour MPs have called for him to go.

At a key meeting on Tuesday, Labour-backing unions were divided over whether to call for Starmer to set out a timetable for his departure, the Guardian understands, but privately agreed they did not feel he should lead the party into the next election.

Sources also told the Guardian that Ed Miliband, who has privately suggested to Starmer that he should consider setting out a timeline for his departure, had been prepared to run for leader himself if Streeting had gone over the top.

Senior Labour figures said Miliband, the energy secretary, who allies say saw himself as kingmaker rather than leadership contender, would have struck a deal with Angela Rayner and Louise Haigh, chair of the Tribune group, for one of them to run as the soft-left candidate, in the event Burnham did not.

Earlier on Tuesday, a defiant Starmer told his cabinet he would fight on as prime minister after a turbulent few days in the wake of crushing election results across Britain last week, saying the threshold for a leadership challenge had not been met.

A series of cabinet ministers, including David Lammy, the deputy prime minister, rallied round their embattled leader, while more than 110 different backbenchers signed a letter saying it was not the time for a challenge.

“I take responsibility for these election results and I take responsibility for delivering the change we promised,” Starmer told the cabinet meeting. “The past 48 hours have been destabilising for government and that has a real economic cost for our country and for families. The Labour party has a process for challenging a leader and that has not been triggered.

“The country expects us to get on with governing. That is what I am doing and what we must do as a cabinet.”

Sources told the Guardian that Starmer did not give cabinet critics time to respond, before moving the conversation on to the Middle East.

There was anger within cabinet ranks over what they regarded as Streeting’s attempts to destabilise Starmer, after three of his closest allies – the ministers Jess Phillips, Zubir Ahmed and Alex Davies-Jones – resigned from government and called on the prime minister to go.

The health secretary had previously insisted he would not initiate a contest, but would join one if it was already taking place. “Wes has got a brass neck,” one minister said. “Some colleagues are furious with him. There were evil looks in his direction.”

While several sources close to Streeting suggested he did not have the numbers to launch a challenge, one ally said it would be premature to say the heath secretary would not be able to do so. “Don’t rule it out yet,” they said.

Miatta Fahnbulleh, the first minister to resign on Tuesday, is backing Burnham to replace Starmer. She told the Guardian she wanted the prime minister to set out a timetable for his departure that would allow the Greater Manchester mayor to run.

“Any contest we have had to have out strongest players in it,” she said. “We are not the Tories, we are only going to do this once. We need to have a proper process that allows all candidates to set out their vision for the country.”

However, Burnham’s hopes of returning to Westminster were dealt a blow as the Merseyside MP Marie Rimmer, whose seat had been named by key allies for a potential byelection, said she would not stand down and backed Starmer to stay.

Starmer was hoping that his second king’s speech, which will be delivered by King Charles on Wednesday, will be a further reset moment for the government that will help unite his deeply divided party.

While he appears to have survived for now, even his most loyal ministers acknowledge that he is unlikely to take Labour into the next election, unless he can dramatically turn round his and the government’s fortunes.

In a sign of how close to the brink Starmer may have come, his home secretary, Shabana Mahmood, spent much of the day on resignation watch. Eventually, her spokesperson told reporters: “No [she will not resign]. She is getting on with the job.”

Lammy urged MPs trying to get rid of Starmer to back off. “Let’s just step back, take a breath, let’s remember that we have the king’s speech, we are in government to do a job of work,” he said.

“It’s been 24 hours now and nobody has come forward to put themselves forward in the processes that exist in the party. No one seems to have the names to stand up against Keir Starmer.

“And for those who are suggesting that he should stand down, they should say which candidate would be better. Let’s get on with the business of running this country and government.”

The Guardian understands that Lammy was among four senior cabinet ministers – with Mahmood, defence secretary John Healey and Yvette Cooper, the foreign secretary – who spoke to Starmer on Monday about his future.

A Labour leadership election is triggered only if 20% of MPs – in this case 81 – nominate a specific candidate to stand against the leader. The party’s ruling NEC would then make arrangements for a full vote of party members.

Embattled FBI director Kash Patel has denied under oath allegations of excessive drinking and unexplained absences on the job, dismissing them as “baseless” during a fiery congressional hearing.

Patel sought to push back as Democrats challenged him over the “extremely alarming” reports, which ran in the Atlantic, which they argued would a mount to a “gross dereliction” of duty.

The FBI director has sued the magazine, and the author of a story it published, filing a defamation lawsuit in US district court for the District of Columbia that seeks $250m in damages.

“It’s a total farce. I don’t even know where you get this stuff,” Patel told Chris Van Hollen, ranking member of the Senate appropriations subcommittee, after he asked about the claims reported by the Atlantic. “I will not be tarnished by baseless allegations.”

When Van Hollen asked if Patel would be willing to take a test to determine whether he has a drinking problem, the FBI director snapped that he would – provided the senator would take it alongside him.

In his opening remarks, Van Hollen said: “What we are learning about what’s happening at the FBI is anything but normal. When your private actions make it impossible for you to perform your public duties, we have a big problem … these reports about your conduct, including reports you’re being so drunk and hungover that your staff had to force entry into your home are extremely alarming, if true, they demonstrate a gross dereliction of your duty and a betrayal of public trust.”

The Atlantic reported that Patel’s alcohol consumption had become “a recurring source of concern across the government”, citing interviews with more than two dozen people, including current and former FBI officials.

The most serious allegations include that his security detail on at least one occasion struggled to rouse him because he appeared intoxicated, and that agents had sought “Swat-level breaching equipment” to gain access to a room where he was unresponsive behind a locked door.

Patel has forcefully denied the allegations. “Print it, all false, I’ll see you in court – bring your checkbook,” he told the magazine. The Atlantic has said it stands by its allegations.