Proprietary memory cards were a weakness of the PSP - you can easily work around it now but at the time you were required to buy expensive Sony Memory Sticks.
The UMDs were ultimately a weakness I feel. They came with the promise of more storage capacity but the load times, noisy loading and shorter battery life that came from them really worked against the system.
The thumbstick was definitely a weakness too - the 'nub' is not great, and the console tried to replicate a lot of PS2 era games whilst missing both a second analogue stick and 2 of the shoulder buttons, which feels quite short sighted when console quality was the aim.
Nowadays, I think hardware reliability is an issue. The system has a lot of delicate parts for a handheld of the era and I've gone through a couple now. Screens and batteries needing replacement is common, and the thumbsticks needing replacement is practically a given at this point I feel. The large for the time screen is prone to being scratched too as it has no time of cover or clamshell design like the GBA SP or DS.
It was a bit of a meme, but I remember a weakness discussed at the time being the tendency for the original PSP to launch your games across the room when you opened it due to the spring loaded disc tray. The 2000 onwards has a tray you open manually
In terms of strengths - I think the longevity of the library is a big one. PSP got games for a long time, somewhat helped by the specs being close enough to both the PS2 and Wii that ports weren't out of the question. Both of those systems being so long lived and popular didn't hurt the PSP, for sure.
Another strength of the system was probably it's power. It blew the GBA out of the water in capability, and far surpassed what the DS was capable of too. The PSP wasn't too dissimilar to a 6th gen console like the PS2 or Dreamcast overall, if not quite on the same level.
Support from big names was a strength too. Sony and third parties really commited to the machine with most of the PS2's big name IPs of the time appearing - Ratchet and Clank, Jak and Daxter, God of War, Gran Turismo, Final Fantasy, Metal Gear Solid, Tekken. It's night and day compared to the support the later PS Vita enjoyed.
Adding to the above post - another benefit of the PSP2000 and PSP3000 is that they offer video output using a component cable. This isn't possible on the 1000.
I sound negative maybe with the systems weaknesses, but in retrospect I actually think the PSP hardware is pretty bad. The software library is worth struggling through, but I don't like the actual machinery at all!