When France won the 2018 World Cup, the Champs Elysee was overflowing with frenzied fans. But more than 2,600 miles away from Paris, there was a much smaller, but no less enthusiastic celebration happening on the small Rue Maréchal Foch. Inside of the one sports bar on the island of Saint Pierre, French flags were everywhere, waved and worn triumphantly, and used to dry the occasional tear. It was a proud moment for all French citizens, especially for the 6,000 who live in a place most people forgets is France. "We are resolutely French," says Stéphane Lenormand, who serves as the president of the Territorial Council of Saint Pierre and Miquelon, a small French territory which – based on its position on the map – might lead most people to assume its part of Canada. While much of North America was once New France, today these islands are the last bits left. Google The French came for the fish. "Saint Pierre exists, and is still French today, because of cod," said Lauriane Detcheverry, the assistant director of L'Arche Museum, which tells the story of the Saint Pierre and Miquelon. "It's really the main reason why people would come here. We have nothing else. I mean, if you look at the island, it's barren. Almost no trees. You cannot grow that much things here. It was one of the richest place to fish cod in the world here. So, you know, you just had to have a boat, and there you go!" Downstairs at the museum, there's an old guillotine. "Something that can prove that you are really in France here is the presence of a guillotine," said Detcheverry. "Guillotine was the only legal way to execute someone in France." "You can't shoot them?" asked correspondent Conor Knighton. "Only chopping off their head?" "Exactly." The 2000 Juliette Binoche film, "The Widow of Saint Pierre," tells the story of the only time the guillotine was ever used in North America. The islands didn't have own, so after a murder conviction in 1889, they asked to borrow a guillotine from Martinique thousands of miles away. For months and months, the convicted prisoner just hung out and waited for the weapon that was allowed to kill him. Over the years, the islands have had quite a violent history. "We changed hands many, many times," said Detcheverry. "Quite often, the city was destroyed and the population was deported." For more than a century, the islands passed back and forth between France and England. But the French kept coming back, and in 1816 they settled for good. "In a way, Saint Pierre is the New World. It was a place where you could become rich," Detcheverry said. The most lucrative era in Saint Pierre's history came entirely by accident. During Prohibition, Saint Pierre became a bootlegging hot spot thanks to its proximity to the United States. "Prohibition was the best thing that happened to us," Detcheverry laughed. "Really. You know, we did nothing to have it. But it was such a good idea for us. People made so much money." The island of Saint Pierre. CBS News In more recent years, the economy has struggled. The once plentiful cod started to disappear, and in 1992 a moratorium on cod fishing in the North Atlantic all but shut down the once-booming fisheries. "It was a very difficult time for the territory," said President Lenormand. "It took almost 20 years to recover and to rebuild. Today we have a fishing sector, but we are developing tourism." Tourists Richard Baum and Cheryl Rodness came to visit Saint Pierre from New Jersey. It so happens that logistically, it's easier to get from New Jersey to Paris than to Saint Pierre. Gallic pride at a sports bar in Saint Pierre, an island off the coast of Newfoundland that is actually a territory of France. CBS News For Americans, a trip to Saint Pierre and Miquelon can involve inconvenient flights and ferries, but the couple thought it would be a blast to spend Bastille Day – France's National Holiday – in a place so unexpectedly French. Here, the wine flows freely. There's dancing and face painting. The locals play games of Pétanque. Everything from the sausages to the ice cream is paid for in Euros. And any lingering doubt that you are, in fact, standing in France is gone the moment you step into one of the bakeries. The fresh baguettes and cakes are just as delicious as the ones you'd see sold in Marseille or Bordeaux. For locals, this is their way of life. For visitors, it's why they come to Saint Pierre and Miquelon. It's a chance to get an authentic taste of France on this side of the Atlantic. More from Conor Knighton's "Island Hopping" series: For more info: © 2019 CBS Interactive Inc. All Rights Reserved.